Papers with automated metrics
Designing Precise and Robust Dialogue Response Evaluators (2020.acl-main)
Copied to clipboard
| Challenge: | Existing automated dialogue response evaluators have only moderate correlation with human judgement and are not robust. |
| Approach: | They propose to build a reference-free dialogue response evaluator that exploits the power of semi-supervised training and pretrained (masked) language models. |
| Outcome: | The proposed model achieves strong correlation with human judgement and generalizes robustly to diverse responses and corpora. |
Generating a Common Question from Multiple Documents using Multi-source Encoder-Decoder Models (D19-56)
Copied to clipboard
| Challenge: | Ambiguous user queries can result in multiple topics being retrieved from search engines. |
| Approach: | They propose a task of generating a common question from multiple documents by training an RNN-based single encoder-decoder generator from document pairs and then a model that aggregates these word distributions to generate a question. |
| Outcome: | The proposed model significantly outperforms existing models when evaluated using automated metrics and human judgments on the MS-MARCO-QA dataset. |
Dual-Axis Generative Reward Model Toward Semantic and Turn-taking Robustness in Interactive Spoken Dialogue Models (2026.acl-long)
Copied to clipboard
Yifu Chen, Shengpeng Ji, Zhengqing Liu, Qian Chen, Wen Wang, Ziqing Wang, Yangzhuo Li, Tianle Liang, Zhou Zhao
| Challenge: | Reinforcement learning (RL) has improved text- and vision-language models, but its application in SDMs is hindered. |
| Approach: | They propose a dual-axis Generative Reward Model that provides semantic quality and interaction timing for SDMs. |
| Outcome: | The proposed model achieves state-of-the-art performance on interaction-quality assessment across a wide spectrum of datasets. |
DISPUTool 3.0: Fallacy Detection and Repairing in Argumentative Political Debates (2025.acl-demo)
Copied to clipboard
| Challenge: | DISPUTool 3.0 is a web-based application for identifying and fixing fallacious arguments in political debates. |
| Approach: | They propose a web-based application designed to identify and repair fallacious arguments in political debates. |
| Outcome: | The proposed tool is based on the ElecDeb60to20 dataset covering US presidential debates from 1960 to 2020. |
Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysis (2024.eacl-long)
Copied to clipboard
Zongxia Li, Andrew Mao, Daniel Stephens, Pranav Goel, Emily Walpole, Alden Dima, Juan Fung, Jordan Boyd-Graber
| Challenge: | Existing evaluation metrics such as coherence and coherency are inadequate for neural topic models. |
| Approach: | They conduct the first evaluation of neural, supervised and classical topic models in an interactive task-based setting. |
| Outcome: | The proposed model performs better on cluster evaluation metrics and human evaluations than classical models on real-world tasks. |
Automatic Evaluation of Local Topic Quality (P19-1)
Copied to clipboard
Jeffrey Lund, Piper Armstrong, Wilson Fearn, Stephen Cowley, Courtni Byun, Jordan Boyd-Graber, Kevin Seppi
| Challenge: | Topic models are evaluated with global topic distributions but without local topic assignments. |
| Approach: | They propose a task to elicit human judgments of token-level topic assignments . they propose to use global metrics to evaluate topic models at a local level . |
| Outcome: | The proposed task elicits human judgments of token-level topic assignments . global metrics agree poorly with human assignments, the authors show . |
FaithDial: A Faithful Benchmark for Information-Seeking Dialogue (2022.tacl-1)
Copied to clipboard
| Challenge: | a new benchmark for hallucination-free dialogues is based on knowledge-based conversational models that generate unsupported utterances . a recent study shows that models that are trustworthy generate unverifiable or factually incorrect statements . |
| Approach: | They propose a data-centric solution to edit hallucinated responses in the Wizard of Wikipedia benchmark. |
| Outcome: | The proposed model improves on the Wizard of Wikipedia benchmark while maintaining engaging conversations. |
Poor Man’s Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference (2023.eacl-main)
Copied to clipboard
Vilém Zouhar, Shehzaad Dhuliawala, Wangchunshu Zhou, Nico Daheim, Tom Kocmi, Yuchen Eleanor Jiang, Mrinmaya Sachan
| Challenge: | State-of-the-art machine translation quality estimation systems have been achieving remarkable correlations with human judgements yet they require human annotations, which are expensive and computationally heavy. |
| Approach: | They propose a problem where one predicts automated metric scores without the reference. |
| Outcome: | The proposed model can estimate automated metrics at the sentence-level without the reference. |
Target-Guided Dialogue Response Generation Using Commonsense and Data Augmentation (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for target-guided response generation are inconsistent with human judgement ratings. |
| Approach: | They propose a technique that finds a bridging path between the source and target and uses it to generate transition responses. |
| Outcome: | The proposed technique outperforms baselines on target-guided response generation task. |
On the Effectiveness of Automated Metrics for Text Generation Systems (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation methods lack a sound theoretical foundation for evaluation campaigns . imperfect automated metrics and insufficiently sized test sets are some of the factors that cause uncertainty. |
| Approach: | They propose a theoretical framework that incorporates different sources of uncertainty, such as imperfect automated metrics and insufficiently sized test sets. |
| Outcome: | The proposed model can be leveraged to improve evaluation protocols regarding reliability, robustness, and significance of the evaluation outcome. |
Multi-Hypothesis Machine Translation Evaluation (2020.acl-main)
Copied to clipboard
| Challenge: | Reliably evaluating Machine Translation (MT) through automated metrics is a long-standing problem. |
| Approach: | They propose to use MT models to generate multiple diverse translations and use them as surrogates to reference translations to obtain a quantification of translation variability. |
| Outcome: | The proposed approach improves correlation with human judgements of quality by 15%. |
RSTGen: Imbuing Fine-Grained Interpretable Control into Long-FormText Generators (2022.naacl-main)
Copied to clipboard
| Challenge: | Using a framework based on Rhetorical Structure Theory, we aim to improve the cohesion and coherence of long-form text generated by language models. |
| Approach: | They propose a framework that utilises Rhetorical Structure Theory to control the discourse structure, semantics and topics of generated text. |
| Outcome: | The proposed framework performs competitively against existing models while offering significantly more controls over generated text than alternative methods. |
Attribute Controlled Dialogue Prompting (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompt-tuning is an increasingly popular parameter-efficient method for adapting large pretrained language models to downstream tasks. |
| Approach: | They propose an instance-specific prompt-tuning algorithm for dialog generation that generates prompts based on instance-level control code rather than the conversation history. |
| Outcome: | The proposed prompt-tuning module is a fraction of the size of the pretrained language model and saves memory and expensive storage space. |
Dictionary-Aided Translation for Handling Multi-Word Expressions in Low-Resource Languages (2024.findings-acl)
Copied to clipboard
| Challenge: | Multi-word expressions (MWEs) are a challenging task in natural language processing . they are defined as combinations of at least two words with distinct lexical, morphological, syntactic, semantic or statistical characteristics. |
| Approach: | They propose a method leveraging an available out-of-context lexicon to improve translations . they propose to use a dictionary-aided translation to better translate multi-word expressions based on human annotations. |
| Outcome: | The proposed method improves translations comparable to those of a human speaker. |
Presentations by the Humans and For the Humans: Harnessing LLMs for Generating Persona-Aware Slides from Documents (2024.eacl-long)
Copied to clipboard
Ishani Mondal, Shwetha S, Anandhavelu Natarajan, Aparna Garimella, Sambaran Bandyopadhyay, Jordan Boyd-Graber
| Challenge: | Existing efforts to automate document-to-slide generation have failed to adapt to the persona of target audience or duration of presentation. |
| Approach: | They propose a concept of end-user specification-aware document to slides conversion that incorporates end- user specifications into the conversion process. |
| Outcome: | The proposed model can create persona-aware presentations tailored to the persona of target audience and cognitive abilities of target audiences. |
LMUNIT: Fine-grained Evaluation with Natural Language Unit Tests (2025.findings-emnlp)
Copied to clipboard
Jon Saad-Falcon, Rajan Pathe Vivek, William Berrios, Nandita Shankar Naik, Matija Franklin, Bertie Vidgen, Amanpreet Singh, Douwe Kiela, Shikib Mehri
| Challenge: | Using natural language unit tests, language models are costly and noisy, and automated metrics provide only coarse, difficult-to-interpret signals. |
| Approach: | They propose a paradigm that decomposes response quality into explicit, testable criteria and a unified scoring model, LMUnit, which combines multi-objective training across preferences, direct ratings, and natural language rationales. |
| Outcome: | The proposed paradigm significantly improves inter-annotator agreement and enables more effective LLM development workflows. |
Am I Me or You? State-of-the-Art Dialogue Models Cannot Maintain an Identity (2022.findings-naacl)
Copied to clipboard
| Challenge: | State-of-the-art open-domain dialogue models fail to maintain character identity throughout discourse . despite improvements in accuracy and self-contradiction, agents take on the role of interlocutor . |
| Approach: | They formalize and quantify the deficiency in character identity modeling by using human evaluations. |
| Outcome: | The proposed models reduce mistaken identity issues by nearly 65% according to human annotators while improving engagingness. |
Likelihood-based Mitigation of Evaluation Bias in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are widely used to evaluate natural language generation tasks as automated metrics. |
| Approach: | They propose to use LLMs to evaluate sentences with higher likelihoods and lower likelihoods to mitigate the likelihood bias. |
| Outcome: | The proposed method overrates sentences with higher likelihoods while underrating sentences with lower likelihoods. |
Improving Generation and Evaluation of Visual Stories via Semantic Consistency (2021.naacl-main)
Copied to clipboard
| Challenge: | Story visualization is an underexplored task that requires a generative model to generate images . prior work has focused on image generation but there is room for improvement . |
| Approach: | They propose to add a dual learning framework to reinforce semantic alignment between story and generated images and a copy-transform mechanism to model sequentially-consistent story visualization. |
| Outcome: | The proposed models outperform text-to-image synthesis models on the story visualization task . the proposed models also improve visual quality, coherence and relevance . |
Ask to Learn: A Study on Curiosity-driven Question Generation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing work on Question Generation focuses on generating relevant questions given text with an answer . human ability to ask questions goes beyond evaluation of reading comprehension . |
| Approach: | They propose a novel text generation task based on a conversational question-asking dataset . they investigate automated metrics to measure different properties of Curious Questions . |
| Outcome: | The proposed task is based on a conversational Question Answering dataset . the results show that humans tend to ask questions with the goal of obtaining new information . |
Semi-supervised Formality Style Transfer using Language Model Discriminator and Mutual Information Maximization (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Formality style transfer is the task of converting informal sentences to grammatically-correct formal sentences. |
| Approach: | They propose a semi-supervised formality style transfer model that utilizes a language model-based discriminator to maximize the likelihood of the output sentence being formal. |
| Outcome: | The proposed model outperforms state-of-the-art models in terms of automated metrics and human judgement. |
Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity (2020.coling-main)
Copied to clipboard
| Challenge: | End-to-end neural data-totext generation has faced challenges generalizing to new domains and generating semantically consistent text. |
| Approach: | They propose a neural data-to-text generation system that makes minimal assumptions about the data representation and target domain. |
| Outcome: | The proposed system achieves state of the art results on four major D2T datasets with better semantic fidelity than the state-of-the-art methods. |
Pedagogically Aligned Objectives Create Reliable Automatic Cloze Tests (2024.naacl-long)
Copied to clipboard
| Challenge: | a recent study has focused on the cloze training objective of Masked Language Models . distractors must be distinct and incorrect, and can be biased if the test creator is testing two versions of a text . |
| Approach: | They propose a method that jointly optimizes sets of distractors from Masked Language Models. |
| Outcome: | The proposed method has stronger correlation with teacher-created comprehension tests than state-of-the-art neural method and is more internally consistent. |
Changing the Mind of Transformers for Topically-Controllable Language Generation (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing interactive writing assistants do not allow authors to guide text generation in desired topical directions. |
| Approach: | They propose a framework that displays multiple candidate upcoming topics and generates a text generation model that adheres to the chosen topics. |
| Outcome: | The proposed model generates fluent sentences related to the selected topics, as judged by automated metrics and crowdsourced workers. |
FIREBALL: A Dataset of Dungeons and Dragons Actual-Play with Structured Game State Information (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work shows that large language models that have access to state information can generate higher quality game turns than LLMs that use dialog history alone. |
| Approach: | They present a dataset of game play sessions from real D&D gameplay on Discord with true game state info. |
| Outcome: | The proposed model can generate executable Avrae commands, especially after fine tuning. |
Favi-Score: A Measure for Favoritism in Automated Preference Ratings for Generative AI Evaluation (2024.acl-long)
Copied to clipboard
| Challenge: | Generative AI systems are becoming ubiquitous for all kinds of modalities . evaluation of generated outputs is increasingly difficult due to cost and complexity of human evaluations. |
| Approach: | They propose to evaluate preference ratings on sign accuracy and favoritism . they propose to use automated metrics to assess generated outputs . |
| Outcome: | The proposed evaluations of preference ratings rely on correlation to human judgments or sign accuracy scores, but this does not tell the whole story. |
Benchmarking Direct Preference Optimization for Medical Large Vision–Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large vision-language models (LVLMs) are gaining traction in clinical tasks such as diagnostic support, report generation, and medical question answering. |
| Approach: | They present a systematic evaluation of nine DPO variants applied to two leading medical LVLMs. |
| Outcome: | The proposed model improves alignment and reduces severe hallucinations, but yields inconsistent gains over supervised fine-tuning. |
Elevating Legal LLM Responses: Harnessing Trainable Logical Structures and Semantic Knowledge with Legal Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to large language models focus on semantic similarity, neglecting the intricate logical structures and reasoning essential for addressing complex legal issues. |
| Approach: | They propose a Logical-Semantic Integration Model (LSIM) that bridges semantic and logical coherence and a supervised framework that integrates semantic features with in-context learning. |
| Outcome: | The proposed framework significantly improves accuracy and reliability on a real-world legal QA dataset. |
Dialect-robust Evaluation of Generated Text (2023.acl-long)
Copied to clipboard
Jiao Sun, Thibault Sellam, Elizabeth Clark, Tu Vu, Timothy Dozat, Dan Garrette, Aditya Siddhant, Jacob Eisenstein, Sebastian Gehrmann
| Challenge: | Existing evaluation metrics that are not robust to dialect variation are difficult to measure for many groups of users and can penalize systems for producing text in lower-resource dialects. |
| Approach: | They propose a dialect-robust evaluation metric that produces the same score for system outputs that share the same semantics but are expressed in different dialects. |
| Outcome: | The proposed method significantly improves dialect robustness while preserving the correlation between automated metrics and human ratings. |
Improving Automated Evaluation of Open Domain Dialog via Diverse Reference Augmentation (2021.findings-acl)
Copied to clipboard
| Challenge: | Prior work has shown that having multiple valid references is important for automated evaluations. |
| Approach: | They propose a technique for automatically expanding a human generated reference to a set of candidate references. |
| Outcome: | The proposed method improves correlations between human-generated metrics and human ratings of system outputs. |
Evaluating LLMs for Targeted Concept Simplification for Domain-Specific Texts (2024.emnlp-main)
Copied to clipboard
| Challenge: | Simplifying the entire text makes it understandable but sometimes removes important details. |
| Approach: | They propose a simplification task for rewriting text to help readers comprehend text containing unfamiliar concepts and introduce a dataset of 22k definitions from 13 academic domains paired with a difficult concept within each definition. |
| Outcome: | The proposed model outperforms open-source and commercial models on the task and human judges prefer explanations over simplifications of the difficult concept. |
ProxyQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models (2024.acl-long)
Copied to clipboard
Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Yunlong Feng, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, Linqi Song
| Challenge: | Existing evaluation methods for large language models are labor-intensive and lack efficiency. |
| Approach: | They propose a framework dedicated to assessing long-text generation that includes in-depth human-curated meta-questions spanning various domains . they use a set of proxy-quests with pre-annotated answers to assess the content's quality by incorporating the generated texts as contextual background. |
| Outcome: | The proposed framework assesses the quality of long-text content by matching it with references through human evaluation or automated metrics. |
PAKTON: A Multi-Agent Framework for Question Answering in Long Legal Agreements (2025.emnlp-main)
Copied to clipboard
| Challenge: | Contract review is a complex and time-intensive task that typically requires legal expertise. |
| Approach: | a new open-source contract review framework is designed to handle complexities of contract analysis . PAKTON is a retrieval-augmented generation framework with plug-and-play capabilities . |
| Outcome: | The open-source framework outperforms models in predictive accuracy, retrieval performance, explainability, completeness, and grounded justifications. |
Correction of Errors in Preference Ratings from Automated Metrics for Text Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods are over-confident in assigning significant differences between systems . Currently, the most reliable evaluation methods for text generation are human-based evaluations. |
| Approach: | They propose to combine human ratings with automated ratings to reduce the amount of human ratings needed to arrive at robust results. |
| Outcome: | The proposed evaluation protocol reduces the amount of human ratings by 50% while yielding the same evaluation outcome as the pure human evaluation in 95% of cases. |
UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to improve UI code generation rely on expensive human feedback or distilling a proprietary model. |
| Approach: | They propose to use automated feedback to guide large language models to generate UI code . they use a large synthetic dataset to generate improved models and refine them . |
| Outcome: | The proposed model outperforms baseline models and larger proprietary models . the model outpersforms models with automated metrics and human preferences . |
A Human Evaluation of AMR-to-English Generation Systems (2020.coling-main)
Copied to clipboard
| Challenge: | a recent human evaluation of AMR generation systems is compared to automated metrics. |
| Approach: | They propose a human evaluation which collects fluency and adequacy scores and categorization of error types for AMR generation systems. |
| Outcome: | The results show that human evaluations are more nuanced than automated metrics. |
TLDR: Extreme Summarization of Scientific Documents (2020.findings-emnlp)
Copied to clipboard
| Challenge: | TLDR generation requires expert background knowledge and understanding of complex domain-specific language. |
| Approach: | They propose a learning strategy that exploits titles as an auxiliary training signal. |
| Outcome: | The proposed method improves upon strong baselines under both automated metrics and human evaluations. |
Distribution Aware Metrics for Conditional Natural Language Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing metrics for conditional natural language generation rely on pairwise comparisons between a single generated text and the best-matching reference. |
| Approach: | They propose a family of meta-metrics that build on existing pairwise distance functions to evaluate conditional natural language generation models. |
| Outcome: | The proposed method evaluates the ability of a model to generate text matching diversity in references in visual description and summarization. |
Could you give me a hint ? Generating inference graphs for defeasible reasoning (2021.findings-acl)
Copied to clipboard
| Challenge: | Defeasible reasoning is a mode of reasoning where conclusions can be overturned by taking into account new evidence. |
| Approach: | They propose to automatically generate inference graphs for a defeasible inference task by transfer learning from a related NLP task. |
| Outcome: | The proposed method generates meaningful graphs for a defeasible inference task and human accuracy improves by 20%. |
PersLLM: A Personified Training Approach for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models exhibit human-like intelligence, enabling them to simulate human behavior and support various applications that require both humanized communication and extensive knowledge reserves. |
| Approach: | They propose a framework for better data construction and model tuning to unlock the potential of LLM personification by using Chain-of-Thought prompting and anti-induction. |
| Outcome: | The proposed framework improves data construction and model tuning for insufficient data usage and rigid behavior patterns. |
CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells (2025.naacl-long)
Copied to clipboard
| Challenge: | Current review comment evaluation metrics rely on comparisons with a human-written reference for a given code change (also called a diff). |
| Approach: | They propose to use a reference-free metric to measure review quality like conciseness, comprehensiveness, and relevance to compare the quality of code changes with human-written references. |
| Outcome: | The proposed metric can produce fine-grained scores that have the greatest alignment with human judgment and are more sensitive than reference-based metrics. |
Keep it Private: Unsupervised Privatization of Online Text (2024.naacl-long)
Copied to clipboard
| Challenge: | Authorship obfuscation has been evaluated in narrow settings in the NLP literature . superficial edit operations can lead to unnatural outputs, authors say . |
| Approach: | They propose an automatic text privatization framework that fine-tunes a large language model via reinforcement learning to produce rewrites that balance soundness, sense, and privacy. |
| Outcome: | The proposed method maintains high text quality according to automated metrics and human evaluation, and successfully evades several automated authorship attacks. |
GupShup: Summarizing Open-Domain Code-Switched Conversations (2021.emnlp-main)
Copied to clipboard
Laiba Mehnaz, Debanjan Mahata, Rakesh Gosangi, Uma Sushmitha Gunturi, Riya Jain, Gauri Gupta, Amardeep Kumar, Isabelle G. Lee, Anish Acharya, Rajiv Ratn Shah
| Challenge: | Abstractive summarization is the process of generating a condensed version of a given conversation while preserving the most salient aspects. |
| Approach: | They propose to use a dataset to analyze code-switched conversations in Hindi and English to summarize them. |
| Outcome: | The proposed dataset contains over 6,800 code-switched conversations and their corresponding human-annotated summaries in English (En) and Hi-En. |
An Evaluation Framework for Legal Document Summarization (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing metrics for summarizing legal documents fail to evaluate intent in the original text. |
| Approach: | They propose an automated intent-based summarization metric which shows a better agreement with human evaluation as compared to other automated metrics like BLEU, ROUGE-L etc. |
| Outcome: | The proposed method shows that human evaluation is more accurate than other metrics. |
Automatic Reference-Based Evaluation of Pronoun Translation Misses the Point (D18-1)
Copied to clipboard
| Challenge: | a range of issues limit the performance of the automated metrics. |
| Approach: | They propose to use semi-automatic metrics and test suites instead of fully automatic metrics for pronoun translation. |
| Outcome: | The proposed metrics improve translation accuracy by comparing them against a manually annotated dataset . the proposed metrics are semi-automatic and test suites in place of fully automatic metrics. |
APPLS: Evaluating Evaluation Metrics for Plain Language Summarization (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for plain language summarization (PLS) lack a dedicated assessment metric and the suitability of text generation evaluation metrics is unclear due to unique transformations. |
| Approach: | They propose a granular meta-evaluation testbed to evaluate PLS metrics . they identify four PLS criteria and define perturbations that sensitive metrics should be able to detect . |
| Outcome: | The proposed testbed assesses performance of 14 existing metrics including scores, features, and prompt-based evaluations. |
Language Model Augmented Relevance Score (2021.acl-long)
Copied to clipboard
| Challenge: | Existing metrics that compare the candidate with the human reference do not consider the context, resulting in poor correlation with human judgements. |
| Approach: | They propose a language model-aware metric that augments the human reference while considering the context to provide evaluation scores that correlate highly with human judgements. |
| Outcome: | The proposed metric achieves higher correlation with human reference judgements and differentiates well-formed candidates from adversarial samples to a larger degree. |
So many design choices: Improving and interpreting neural agent communication in signaling games (2023.findings-acl)
Copied to clipboard
| Challenge: | Emergent language games are experimental protocols designed to model how communication may arise among a group of agents. |
| Approach: | They propose to adopt a signaling game in which a sender is exposed to an image and generates a sequence of symbols that is transmitted to a receiver. |
| Outcome: | The proposed language improves when the sender is exposed to an image and generates a sequence of symbols that is transmitted to a receiver. |
Conversing by Reading: Contentful Neural Conversation with On-demand Machine Reading (P19-1)
Copied to clipboard
Lianhui Qin, Michel Galley, Chris Brockett, Xiaodong Liu, Xiang Gao, Bill Dolan, Yejin Choi, Jianfeng Gao
| Challenge: | a new approach to contentful neural conversation is proposed . end-to-end models are effective in learning fluent responses, but their responses are often vacuous and uninformative. |
| Approach: | They propose a model that provides the conversation model with relevant text on the fly as a source of external knowledge. |
| Outcome: | The proposed model improves the informativeness and diversity of generated output compared to previous methods. |
Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations (2023.acl-long)
Copied to clipboard
Lucy Lu Wang, Yulia Otmakhova, Jay DeYoung, Thinh Hung Truong, Bailey Kuehl, Erin Bransom, Byron Wallace
| Challenge: | Prior work has shown that models may exploit shortcuts that are difficult to detect using standard n-gram similarity metrics such as ROUGE. |
| Approach: | They propose to use human-assessed summary quality facets and pairwise preferences to improve MDS evaluation methods. |
| Outcome: | The proposed methods improve the quality of literature review summarization models . they use human-assessed summary quality facets and pairwise preferences . |
Cognitive Reframing of Negative Thoughts through Human-Language Model Interaction (2023.acl-long)
Copied to clipboard
Ashish Sharma, Kevin Rushton, Inna Lin, David Wadden, Khendra Lucas, Adam Miner, Theresa Nguyen, Tim Althoff
| Challenge: | Psychotherapy can help people overcome negative thoughts by replacing them with a more hopeful "reframed thought" but clinician shortages and mental health stigma often limit access to therapy. |
| Approach: | They propose a framework of seven linguistic attributes that can be used to reframe a thought . they use a retrieval-enhanced in-context learning model to generate reframed thoughts . |
| Outcome: | The proposed model is based on a human-centered study of 600 situations, thoughts and reframes on 2,000 mental health websites. |
Revisiting Automated Topic Model Evaluation with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Topic models are an unsupervised dimensionality reduction technique that help organize large text collections. |
| Approach: | They propose to use large language models to evaluate document output and determine optimal number of topics. |
| Outcome: | The proposed model performs better on coherence ratings of word sets than on intrustion detection. |
Self-Attention Architectures for Answer-Agnostic Neural Question Generation (P19-1)
Copied to clipboard
| Challenge: | Neural architectures based on self-attention have attracted interest from the research community . a recent study examined the performance of Transformers on a task of Neural Question Generation . |
| Approach: | They propose to adapt Transformers to a task of Neural Question Generation without constraining the model to focus on a specific answer passage. |
| Outcome: | The proposed architectures have obtained significant improvements over the state-of-the-art in several tasks. |
Pointwise Mutual Information Based Metric and Decoding Strategy for Faithful Generation in Document Grounded Dialogs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing metrics for faithfulness of response are not aligned with human judgments. |
| Approach: | They propose a new metric that utilizes (Conditional) Point-wise Mutual Information (PMI) between the generated response and the source document, conditioned on the dialogue. |
| Outcome: | The proposed metric improves on BEGIN benchmarks and shows that it generates more faithful responses than standard decoding techniques. |
A Call for Standardization and Validation of Text Style Transfer Evaluation (2023.findings-acl)
Copied to clipboard
| Challenge: | Text style transfer (TST) evaluation is inconsistent in practice. |
| Approach: | They conduct a meta-analysis on human and automated TST evaluation and experimentation . they find a standardization gap and a validation gap in the field . |
| Outcome: | The authors find that evaluation procedures are inconsistent and that they need to improve on them. |
An “Integrative Survey on Mental Health Conversational Agents to Bridge Computer Science and Medical Perspectives” (2023.emnlp-main)
Copied to clipboard
| Challenge: | Mental health conversational agents (a.k.a. chatbots) are widely studied for their potential to offer accessible support to those experiencing mental health challenges. |
| Approach: | They review 534 papers on building mental health-related conversational agents . they recommend a few recommendations to bridge the disciplinary divide . |
| Outcome: | The systematic review reveals 136 key papers on building mental health-related conversational agents with diverse characteristics of modeling and experimental design techniques. |
Transparent and Coherent Procedural Mistake Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Procedural mistake detection (PMD) is a problem of classifying whether a human user has successfully executed a task. |
| Approach: | They extend PMD to require generating visual self-dialog rationales to inform decisions . they leverage a natural language inference model to formulate two automated metrics for coherence of generated rationale. |
| Outcome: | The proposed model improves on a reframed task with a natural language inference model and a multi-faceted metrics visualization of common outcomes. |
Story Morals: Surfacing value-driven narrative schemas using large language models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using large language models, we extract and validate story morals across a diverse set of narrative genres. |
| Approach: | They propose a task of narrative schema labelling based on the concept of "story morals" they use large language models to extract and validate story morals across a diverse set of genres . |
| Outcome: | The proposed method extracts and validates story morals across folktales, novels, movies and TV, personal stories from social media and the news using automated metrics and human assessments. |
CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era (2025.findings-acl)
Copied to clipboard
Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma, Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, Jiajun Chen
| Challenge: | Image captioning has been a challenge for vision-language researchers for decades . current VLMs focus on tasks like visual question answering (YA) but image captioning is not as advanced as expected. |
| Approach: | They evaluate VLMs' performance on image captioning using human annotations . they find that some metrics show high caption-level agreement with humans . |
| Outcome: | The proposed model outperforms open-source models on image captioning . it achieves 93.4% correlation with human rankings at $4 per test . |
ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clustering (2025.acl-long)
Copied to clipboard
| Challenge: | Topic models and document clustering evaluations often use automated metrics that align poorly with human preferences or require expert labels that are intractable to scale. |
| Approach: | They propose a protocol for evaluating topic models and document clustering evaluations that uses crowdworker annotations to validate automated proxies. |
| Outcome: | The proposed protocol is scalable and easy to adapt to an LLM prompt. |
Revisiting Automated Evaluation for Long-form Table Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing automated metrics for long-form table question answering (LFTQA) are poorly correlated with human judgments and fail to distinguish between factually accurate responses and those that are factual incorrect. |
| Approach: | They propose to use a meta-evaluation dataset to assess the effectiveness of LLM-based LFTQA systems. |
| Outcome: | The proposed meta-evaluation dataset includes 2,988 human-annotated examples. |
ByteSized32: A Corpus and Challenge Task for Generating Task-Specific World Models Expressed as Text Games (2023.emnlp-main)
Copied to clipboard
| Challenge: | We show that language models can generate explicit, interpretable, and interactive world models of scientific and common-sense reasoning tasks. |
| Approach: | They propose a corpus of 32 reasoning-focused text games expressed as hundreds of lines of Python code to facilitate this task. |
| Outcome: | The proposed games can generate runnable games on unseen topics in 28% of cases. |
CLEAR: A Clinically Grounded Tabular Framework for Radiology Report Evaluation (2025.findings-emnlp)
Copied to clipboard
Yuyang Jiang, Chacha Chen, Shengyuan Wang, Feng Li, Zecong Tang, Benjamin M. Mervak, Lydia Chelala, Christopher M Straus, Reve Chahine, Samuel G. Armato Iii, Chenhao Tan
| Challenge: | Existing metrics lack the granularity and interpretability to capture nuanced clinical differences between candidate and ground-truth radiology reports. |
| Approach: | They propose a tabular framework with E**xpert-curated labels and an attribute-level comparison for radiology report evaluation (**CLEAR) |
| Outcome: | The proposed framework can extract clinical attributes and provide automated metrics that are strongly aligned with clinical judgment. |
MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent smaller language models rely on synthetic data generated using larger Language models. |
| Approach: | They propose a method for generating synthetic data that enhances diversity through meta-prompting . they use 25 million tokens of synthetic data generated by a language model orchestrated by multiple “expert” LLM agents to collaboratively generate data. |
| Outcome: | The proposed method outperforms the base LLM in Finance and Biomedicine with 25 million tokens of synthetic data. |
What’s the Difference? Supporting Users in Identifying the Effects of Prompt and Model Changes Through Token Patterns (2025.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation methods for prompting for large language models have limitations such as being labor-intensive or lacking insights. |
| Approach: | They propose a new approach that automatically distinguishes between random variations and systematic differences in language model outputs by using token patterns. |
| Outcome: | The proposed method combines both automation and human analysis to provide new insights into established prompt data. |
PhoniTale: Phonologically Grounded Mnemonic Generation for Typologically Distant Language Pairs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Vocabulary acquisition is a challenge for second-language learners when learning typologically distant languages such as English and Korean, where phonological and structural mismatches complicate vocabulary learning. |
| Approach: | They propose a cross-lingual mnemonic generation system that performs IPA-based phonological adaptation and syllable-aware alignment to retrieve L1 keyword sequence and uses LLMs to generate verbal cues. |
| Outcome: | The proposed system outperforms human-written and automated mnemonics in a short-term recall test with human participants and achieves quality comparable to human-writing mnms. |
Text Style Transfer Evaluation Using Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have demonstrated their capacity to match and even exceed average human performance across diverse, unseen tasks. |
| Approach: | They compare the results of different LLMs in TST evaluation using multiple input prompts and introduce the concept of prompt ensembling. |
| Outcome: | The proposed model outperforms human evaluations on multiple input prompts. |
Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs (2025.acl-long)
Copied to clipboard
Sumanth Doddapaneni, Mohammed Safi Ur Rahman Khan, Dilip Venkatesh, Raj Dabre, Anoop Kunchukuttan, Mitesh M Khapra
| Challenge: | Evaluating machine-generated text remains a challenge in NLP for non-English languages . current evaluation frameworks focus on English, revealing a gap in multilingual evaluations . |
| Approach: | They propose a cross-lingual auto evaluation framework that includes evaluator LLMs and a test set specifically designed for multilingual evaluation. |
| Outcome: | The proposed model aligns more closely with human judgments than proprietary models on non-English language evaluations. |
Language Models can Subtly Deceive Without Lying: A Case Study on Strategic Phrasing in Legislation (2025.acl-long)
Copied to clipboard
Atharvan Dogra, Krishna Pillutla, Ameet Deshpande, Ananya B. Sai, John J Nay, Tanmay Rajpurohit, Ashwin Kalyan, Balaraman Ravindran
| Challenge: | blatant lying or unintentional hallucination are common in large language models. |
| Approach: | They build a testbed mimicking a legislative environment where a corporate lobbyist module is proposing amendments to bills that benefit a specific company while evading identification by strong LLM detectors. |
| Outcome: | The proposed model can be used to detect deception in legislative environments and to optimize its phrasing to avoid detection by strong detectors. |
Not What the Doctor Ordered: Surveying LLM-based De-identification and Quantifying Clinical Information Loss (2025.emnlp-main)
Copied to clipboard
Kiana Aghakasiri, Noopur Zambare, JoAnn Thai, Carrie Ye, Mayur Mehta, J Ross Mitchell, Mohamed Abdalla
| Challenge: | De-identification is an application of NLP where automated algorithms remove identifying information of patients and providers. |
| Approach: | They propose to use generative large language models to de-identify patients and providers . they propose to validate existing metrics to quantify extent of inappropriate removal . |
| Outcome: | The proposed method is based on a survey of LLM-based de-identification research . it shows that the models perform poorly in identifying clinically relevant changes . |
UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic Languages (2026.acl-long)
Copied to clipboard
Pranjal A Chitale, Varun Gumma, Sanchit Ahuja, Prashant Kodali, Manan Uppadhyay, Deepthi Sudharsan, Sunayana Sitaram
| Challenge: | Developing culturally grounded multilingual AI systems is challenging for low-resource languages . synthetic data is underexplored, but its effectiveness in multilingual and multicultural contexts is understudied . |
| Approach: | They propose a top-up synthetic data generation framework grounded in Wikipedia content . they use 9.5M data points across 13 Indian languages and English to generate a high-quality dataset . |
| Outcome: | The proposed model improves on NLG tasks and narrows performance gaps with high-resource languages. |
Instruction-Guided Poetry Generation in Arabic and Its Dialects (2026.findings-acl)
Copied to clipboard
Abdelrahman Sadallah, Kareem Elozeiri, Mervat Abassy, Rania Elbadry, Mohamed Anwar, Abed Alhakim Freihat, Preslav Nakov, Fajri Koto
| Challenge: | Existing literature on Arabic poetry has focused on analysis tasks such as interpretation or metadata prediction, e.g., rhyme schemes and titles. |
| Approach: | They propose to use a large-scale instruction-based dataset to generate Arabic poetry based on predefined criteria such as style and rhyme . |
| Outcome: | The proposed model can generate poetry that is aligned with user requirements, based on automated metrics and human evaluation with native Arabic speakers. |